Development of Interactive Multimedia Applications 

N 9 3 - £2^2-1 

Albert Leigh 1 ^ 

Lui Wang n j ^ o 

Software Technology Branch/PT4 
NASA Lyndon B. Johnson Space Center C j 

Houston, TX 77058 / 


ABSTRACT 

Multimedia is making an increasingly significant contribution to our 

informational society. The usefulness of * ,s ' s . 

in education business presentations, informational kiosks t 8; 
museums) training and the entertainment environment. Institutions, from 
grade schools to medical schools, are exploring the use of multifaceted 
electronic text books and teaching aids to enhance course materials. 
Through multimedia, teachers and students can take full advantage of the 
cognitive value of animation, audio, video and other ^ 

seamless application. The Software Technology Branch at NASA Johnson 
S D ace Center (NASA/JSC) is taking similar approaches to apply the state- 
of-the-a^ Technology io space training, mission operations and other 
applications. This paper discusses the characteristics and development o 
multimedia applications at the NASA/JSC. 


1.0 INTRODUCTION 

rrmUrnif-dia technology is changing the way computers are applied. For instance, 
wlttdie^die^resotm^ form ^f i^ns^mrsTmamriais, or dme^ alTmust be 

and draw closer attention than traditional systems. 

All computer manufacturers support multimedia capabilities in one form or another, 
rvimnanies like Apple Computers and Microsoft Corporation have released multime 
cZb^Zr^ ng ^sy^msTd Graphical User Interface (GUI) environments. Many ojer 
Xw^e ^ hardware vendors have also launched products for these environments that 
comply to the standards defined by international and professional organizations. Through 
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these non-proprietary standards, products have become more homogeneous and can be 
utitized over a large sector. 

In this paper, the characteristics and components of multimedia environments will be 
discussed. A summary of multi-platform development activities of interactive multimedia 
applications at the NASA/JSCs Software Technology Branch (STB) will be reported. The 
final section outlines the experiences gained from developing these projects. 


2.0 MULTIMEDIA ENVIRONMENT 

As the term implies, multimedia is the integration of several media in an interactive 
computing environment (Figure 2.1). Conventional media include audio, video, still- 
photographs and printed documents, whereas computerized media consist of test, graphics 
animation, spreadsheets and navigational links. When combined with the computer's 
interactive capability, audiences need no longer be passive in a multimedia system because 
they can navigate through a maze of information based on an intuitive structure. They 
decide when to start, when to stop, which piece of information to retrieve, and where to go 
for pertinent data. Although the term multimedia is vaguely defined, it is obvious that 
multimedia is used to enhance the way messages are conveyed in a multi-sensory form As 
a picture is worth a thousand words, time-sequenced pictures like video, combined with 
audio, transmit information more effectively. It is one of the most exciting things to 
happen with computers in many years. 



Figure 2.1 A Multimedia Environment 


A multimedia software environment generally consists of two levels: viewing and 
authoring. Viewing refers to accessing a pre-composed system interactively without the 
capabdity to make modifications. Authoring involves organizing, mixing, and matching 
dirlerent forms of media; generating links for navigation; designing user-interfaces; and. 


218 




creating error handling routines. Scanning devices are used to scan in photographs and flat 
documents, while appropriate digitizing devices are used to convert audio and video data 
from analog to digital forms. Of all the included media, video is the most elaborate, 
complicated, and intriguing. It is composed of sequences of video frames or images which 
are stored and replayed in rapid succession. The National Television and Standard 
Committee (NTSC) format used in the United States displays thirty frames per second. 
Thus video documents cpntaining a wealth of information, such as sound and still and 
motion images, demand greater system resources. Many computer hardware and software 
companies, including consumer electronic companies, have released desktop multimedia 
products with the necessary components and capabilities. 

2.1 Trends 

Available components include single chip solutions for video encoding (i.e. frame 
grabbing) display, and compression/decompression (CODEC), respectively^ With these 
components, high-resolution displays are common to today’s computers. They can now 
display up to 1280 by 1024 resolution and show 24-bit color photo-realistic images, at a 
fraction of the cost of earlier high-priced graphics workstations. Also, a single video 
overlay board (e.g. NTSC to VGA) can display a live television window on a computer, 
and can capture and compress video data on a hard disk. As a result, these capabilities 
allow software vendors to develop and market authoring tools (e.g. Auuiorware s 
Authonvare), non-linear video editing software packages (e.g. Adobe System s Premiere), 
and video special effects systems (e.g. NewTek’s Video Toaster). In addition, mass media 
companies are trying to establish their presence in the new media frontier. For instance, the 
Wall Street Journal reported that Turner Broadcasting System, Inc. is creating interactive 
news documentaries using the company's Cable News Network (CNN) footage and 
interactive games based on characters from its Hanna-Barbera film library to be used on 
CD-ROMs, i.e. compact disk-read only memory. As shown in Table 2.1, many similar 
titles ranging from children books to instructional materials are already on the market 



Amanda Stories 


Beethoven: Symphony 
No. 9 


I Photograph to 
Remember 


Columbus: Encounter, 
Discovery and Beyond 
Microsoft Bookshelf for 
Windows 


The Virtual Museum 


The Madness of Roland 


escnption 


Children adventures and stories 


Interactive classical music 
entertainment and lesson 


A still image essay the works of Pedro 
Meyer, a renowned photographer 


An interactive multimedia lesson of 
Columbus’ voyage to the new world 
A reference library includes an 
encyclopedia, a dictionary, a thesaurus, 
a world alamanac, an atlas and two 

books of quotations 

An interactive electronic museum 
where users can move from room to 
room and select any exhibit for more 

detailed examination 

The world's first hypermedia novel 
with digital video 


stem 



Table 2.1: Commercial multimedia publications 

As these data types demand increasing storage space, manufacturers have responded with 
winchester (hard) disk drives that are smaller in physical size, but greater in capacity. It is 
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not uncommon for new PCs to have 2 billion bytes (GB) of storage. Distribution medium 
for developed applications also needs to be more efficient and economical. CD-ROM, for 
example, can store up to about 650 million bytes (MB), and yet, each disk costs as little as 
a box of floppy disks to produce. The greater the number of disks produced, the less 
expensive per disk is the production cost. In comparison, a box of ten diskettes can store 
approximately 15 MB. Then, there are magneto-optical drives (~1 GB), 3.5 inch floptical 
drives (-120 MB), 8 mm backup tapes (2.3 GB), and digital audio tape (DAT) (8 GB). 

It is now obvious that the multimedia industry has enjoyed an exponential growth within 
the last few years. In fact, it is considered to be one of the fastest growing industries. 
However, the growth could have been much greater if there were open standards for this 
technology. A lack of standards has resulted in proprietary systems that are costly and 
incompatible. This is the main reason that multimedia technology did not reach the 
mainstreams until recently. 

So, what standards exist today, and what are needed in the future? The raster display 
technology of the past decade is one of the forerunners to the current multimedia realm. 
Digital audio has also become a common data type on computer systems from Apple 
Computer, Commodore Computers, Sun Microsystems and PC vendors. Most 
importantly, as of 1992, video data is becoming more common on systems like Macintosh 
and PCs. This is primarily due to the fact that there are currently three non-proprietary 
digital video CODEC standards: Joint Photograph Experts Group (JPEG) standard for still 
image compression; the Consultative Committee on International Telephony and 
Telegraphy (CCITT) Recommendation H.261 for video tele-conferencing; and, the Moving 
Pictures Experts Group (MPEG) for full-motion video compression on digital storage 
media (DSM). Table 2.2 lists the experimental compression ratios that JPEG variant 
coders can achieve and the resulting image quality [2]. 
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22:1 

1.5 

Near original 

11:1 

8 

Lossless JPEG 

^ 2:1 


Table 2.2: JPEG Compression Ratios 

Intel Corporation is one of the pioneers in the digital video compression arena. Intel's 
Digital Video Interactive (DVI) provides powerful CODEC capabilities with a 
programmable processor. DVI allows application developers to choose CODEC algorithms 
for better image quality; more simultaneous video operations, such as scaling motion video 
to a re-sized window or increased flexibility to support special needs of embedded 
applications; or any combination of the above. A key new feature of the components is 
their ability to compress or decompress JPEG images in near realtime. IBM Corporation is 
aggressively supporting this technology in its ActionMedia product line. NewVideo 
Company is another vendor actively developing DVI boards for Apple Macintosh systems. 

Finally, Apple Computers has released a Macintosh system extension called QuickTime that 
allows temporal data types, such as audio and video, to be integrated into applications 
without special hardware. This extension can be used along with the above CODEC 
standards as long as the software interface conforms to QuickTime protocol. With it, video 
messages can be embedded in electronic mail, spreadsheets and presentations. 








3.0 DEVELOPMENT ACTIVITIES 

The use of various forms of media in computer applications has been in place at the 
Software Technology Branch (STB)for several years. However, the applications 
developed today utilize more multimedia capabilities by incorporating audio and video 
which was not feasible in the past. The following sections describe the to-date activities of 
developing multimedia and related applications at NASA/JSC. 

3.1 Multimedia 

The Automated Information Center (AIC) project is an (interactive multimedia torn -of the 
NASA/JSC Information Systems Directorate Products Center (IPC). The IPC is a facility 
dedicated to providing computer-related hardware and software products for sale and loan, 
information searches, product demonstrations, monthly newsletters, andmany ' ot er 
services to the JSC community. The AIC system was developed to provide IPC customers 
with on-demand access to information about the IPC and its products and services. Also, 
to lessen the IPC staffs burden of continually being asked questions by customers about 
general operational topics. By automating this task, the IPC staff will be able to provide 
more individualized assistance to customers requiring detailed mformation, as well as being 
allowed greater freedom to accomplish day-to-day activities. This system was built as a 
fully automated multimedia environment utilizing state-of-the-art personal computer 
components and software in a kiosk-style booth at the IPC facility (Figure 3. ). 



Figure 3.1: Automated Information Center's digital video screen 

The general system configuration includes a generic 486 PC with Microsoft Windows, 
digital CODEC boards, an audio board and a CD-ROM drive. In the application, the user 
is presented with a touch screen interface and menus to invoke general information about 
the IPC and detailed descriptions about the IPC services, products and activities. The user 
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could choose either of these paths from the initial screen. The upper portion of the screen 
contains general navigational buttons. Upon selecting certain buttons, the user receives 
audio and video explanations, schedules of events, software and hardware catalogs, and 
floor plan information. The user can also try out some on-line software or can view 
software demonstration videos. 

Based on this system, STB is exploring potential development of a very large scale 
multimedia information retrieval system for the NASA/JSC's Public Affair Office and the 
U.S. Navy's Informational Survivability Management System (ISMS) for advanced 
damage control. 

3.2 Hypermedia 

A hypermedia product, called Hyperman, was designed and developed by STB. 
Hyperman is a software tool which enables the users of technical manuals to have rapid on- 
line access to documentation with a full range of hypermedia capabilities. With Hyperman 
users can parse documents in their native word processing format and display these 
documents on a UNIX platform employing X-Windows with the Motif Toolkit as shown 
in Figure 3.2. Hyperman is able to display a broad range of media, including text, 
equations, tables, graphics and audio. 



Figure 3.2 Hyperman's on-line documentation 

Hyperman was created with two separate configuration options. The first option creates a 
stand-alone hypermedia tool. The second option creates a "help-system” version of 
Hyperman. In the second configuration. Hyperman can exist within another host software 
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tool. When the user has a need for on-line help. Hyperman can provide context sensitive 
help based on the users current location in the software. 


Hyperman employs a number of techniques to provide for easy transition between the 
user's paper and electronic document domains. These techniques include, electronic 
highlights, electronic notepads, full text search, hyper-links, preservation of the look of the 
K from paper to screen , and easy access to the UNIX environment. This access to the 
UNIX environment means that applications and processes, including the at ^bment of 
video, animation and audio to Hyperman text and graphic objects, ^ St ^ e ^ f ^ 
within Hyperman which will augment the users comprehension. Hyperman s P^ing 
preserves^not only the text and page formats but all of the formats to the text including 
bold-facing, underlining, scripting, subscription, and italics. These capabilmes arc encased 
in a Graphical User Interface (GUI) which attempts to make the task of document 
management, information retrieval and diverse data integration easy. 


3.3 Intelligent Computer-Aided Training (ICAT) 

Expert system technology has been used to develop autonomous training systems for use 
by astronauts, flight controllers and NASA engineers in learning to perform a wide range 
of procedural tasks. STB has developed many ICAT systems based on this technology. 
The Payload Deployment ICAT (PD/ICAT) and Center foformation Systems/Computer 
Operations ICAT (CISCO/ICAT) are examples of such applications de ^oped on 
UNIX/X-Windows and Microsoft Windows environments, respectively. The Acti 
Thermal Control System (ATCS) ICAT for Space Station Freedom tuning was^veloped 
on the Apple Macintosh environment. It incorporates extensive multimedia ru d>e form of 
scanned color photographs, animated graphics, digitized audio, and QuickTime digital 
video to engage the trainee's interest and to present a variety of concepts and system 
functions in a simulated real-world environment. 

Audio and video are significant elements in many subject areas. Flight controllers rely 
heavily on audio information received via their headphones, and astronauts must train for 
many tasks requiring visual inspection. Yet, audio and video elements have been omitted 
from most ICAT systems because they are system dependent. Integrating Apple s 
QuickTime software into the ICAT development environment offers three major 
advantages: it eliminates the cost of specialized hardware, it speeds development time and 
reduces maintenance costs, and it enables the cross-platform portability of the ICAT 
development environment. 

Using interactive graphics of display screens, high resolution photographs of equipment 
and control panels* and full-motion video of operation and maintenance procedures, ICAT 
systems are being developed to supplement and potentially replace large, expensive part 
task trainers and provide "on-demand" training wherever and whenever it is needed. 


3.4 Virtual Environments 

Virtual Environment technology, which is generally referred to a s virtual reality or 1 

reality has the potential to provide an intuitive human-computer interface of unprecedented 
power* (Figure 3.3). The STB is exploring the use of this technology for training as an 
adjunct to its ongoing work in ICAT. Virtual Environments (VE) can place an individu 
into any scenario that can be copied or imagined. The use of VE for training js obvious. 
Crew members can experience a virtual cockpit or an Extravehicular Activity (EVA) and 
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both develop and be trained in new procedures. Payload specialists can learn to operate 
virtual instruments before they are built or flown. The key to VE is the immersion of the 
user in another world. The ability of VE to cause the suspension of disbelief is a power 
argument for the infusion of VE technology into NASA's training program. 



Figure 3.3: A VR system in use at NASA/JSC 


The STB is actively pursuing a number of applications projects that will utilize virtual 
environment technology. Among these projects are training for Space Shuttle EVA, Space 
Station Freedom operations especially those that are cupola based, and Hubble Space 
Telescope repair and maintenance. In addition, STB is working jointly with Marshall 
Space Flight Center to enable personnel at the two centers to simultaneously share the same 
VE, eliminating the usual constraints of location from joint training activities. Finally 
prototypical science laboratories are being developed to enable students to observe physical’ 
phenomena not available in typical student laboratories. 


4.0 CONCLUSION 

Multimedia technology is in a very dynamic growth period with industry standards being 
defined and technical advancements with hardware occurring in many cases faster than new 
products can be released. Applications are being developed to exercise these available 
technologies. Software is reaching cross-platform capability status with peripheral 
component interface utilities and more extensive capabilities and functions incorporated into 
products. As multimedia system and component costs decrease, these systems will become 
attainable by a wider group for varied implementations. 

Although multimedia is here today, the technology is still in its infancy. Mixing and 
matching peripherals with existing peripherals bring up conflicts and incompatibilities. On 
a distributed network, transferring multimedia data can decrease the network throughput 
and efficiency dramatically. The STB is actively pursuing many avenues to overcome these 
deficiencies. Many multimedia data types are quite portable now. Animation, audio, 
graphics and images, spreadsheets and text data can easily be transported between 
hetrogeneous platforms. What is needed most at this time is non-proprietary software 
video compression and decompression capabilities such that the applications developed can 
be delivered without 

requiring costly hardware add-ons. More research and development needs to be done to 
incorporate search and indexing, knowledge capture, and higher bandwidth networking for 
distributed capabilities. 

As important as the hardware and software technology is the skill level of an integrated 
development team. This team of creative professional technical and artistic people from 
various disciplines is necessary for a worthwhile product. With a good design and creative 
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skills, a multidisciplinary effort of computer engineers, graphics artists, a producer or 
manager and other relevant personnel can conquer the risks and barriers It will also result 
in a high-tech masterpiece application and yield high pay-offs. In conclusion, to meet the 
demands of our informational society, the full potential of this revolutionary digital 
technology must be continue to be realized and utilized. 
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